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Improve your reputation! Impress 
the crowd at PC Forum 2004. 
Save the date: March 21-23. 


The conversation starts here. 


BY JEFF UBOIS 


In the online world, identity alone is not enough. We encounter 
strangers on the Internet all the time, and who they are — even their 
“true names” — is hardly enough. We need to know more about people 
when we encounter them without history. What are they known for? 
Who will vouch for them? In short, we need to know their reputations. 


Economically, reputation systems do three major things: They foster 
good behavior, they punish (over the long term) bad behavior when it 
occurs, and they reduce people’s risk of being harmed by others’ bad 
behavior. Oddly, they are the opposite of a traditional insurance sys- 
tem: Rather than sharing risks across a broad population, they assign 
costs directly to those who cause them. People with good reputations 
can charge more or pay less. People with bad reputations shape up, pay 
more (or earn less) or don’t get to do business. (And yes, profit-seeking 
credit-grantors or insurers use reputation to set premiums or allocate 
business.) Because of all these effects, reputation systems make people 
more likely to engage in transactions in the first place, whether that 
means going out on a date or buying a sofa unseen. 


Life really is not a zero-sum game. As any game theorist will tell you, 
games more often have positive outcomes when the players can track 
one another’s behavior. 


Indeed, reputation systems are an excellent illustration of how iterat- 
ing simple rules among actors and over time can produce complex, 
unpredictable results. Reputation systems generally skew positive — 
since the systems that skew negative generally die off. (It’s a lot easier 
to leave a bad website than to leave a bad neighborhood.) All the 
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details, from the incentives for feedback to the incentives the feedback 
creates, matter. 


There are a variety of offline parallels, starting with Santa’s list, but 
online reputation systems — as opposed to, say, a Google search — have 
a specific meaning. 


First, reputation systems concern people and organizations, not things. 
“Reputation is for actors who can change [and are responsible for their 
own] behavior,” says Paul Resnick, an associate professor at the 
University of Michigan's School of Information. “Sony has a reputa- 
tion, but the Sony Vaio Model 123 has a recommendation, rating or 
review.” J.D. Power rates cars; while its ratings may influence the be- 
havior of car makers, the car itself can’t change in response to its rating. 


Second, a reputation adheres to the person or organization and is inde- 
pendent of who’s viewing it. That is, it’s not “personalized” like My- 
Yahoo! or an Amazon collaborative filtering system based on your 
purchase patterns and those of readers with similar patterns. Nor is it a 
friend-of-a-friend network such as LinkedIn or Friendster. We will 
cover these social network systems in our November issue. 


Ironically, Jeff Ubois, who wrote this issue, came to us without a formal 
reputation but through such a social network - through two colleagues, 
Clay Shirky and Jerry Michalski, via multiple links to Release 
1.0/EDventure including a private e-mail list. 


We know him directly now, as he wrote last April’s issue on “digital 
garbage.” When he’s not writing for us, Ubois is an analyst with Ferris 
Research, a San Francisco-based consulting firm specializing in e-mail 
and collaboration technologies. In 1998, he co-founded Omniva, 
which provides software that helps companies secure their e-mail. He 
has been reporting about Internet technologies since 1987. 


— Esther Dyson 
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Some like to think of the Net as a digital village, but in fact it’s closer to a digital city. 
The ability to interact with a billion people on the Net comes with its own costs: 
Dealing with strangers is risky, and verifying their trustworthiness is expensive — 
especially on a case-by-case basis. 


A bigger pool of potential partners and customers implies that a larger fraction of 
our interactions will consist of one-time transactions. With no reason to maintain a 
relationship after that first encounter, trade or tirade, there is a greater temptation, 
ironically, for either truth or deception. Who has not heard — or told — all in a pass- 
ing moment with a total stranger? (When two such strangers meet again it’s often 
worth a movie.) In business, it’s tempting for both parties to provide minimal ser- 
vice or even to commit fraud. Restaurants serving tourists unlikely to return usually 
compare poorly with those that live by repeat business. 


That all changes when people can check each other’s reputation in advance, and 
publicize the quality of any interaction afterwards. When it’s understood that behav- 
ior will be made public, the desire to preserve a good reputation provides an incen- 
tive for both parties to behave honorably. As an interpreted record of past 
performance, reputation is no guarantee of future results, but it’s frequently the best 
predictor of future behavior of a person or company. 


In every society, and in every market, reputation matters. Historically, people have 
killed (or died) to protect or avenge their reputations. It’s through reputation mech- 
anisms that we choose the “best” doctor or lawyer, enlist the community to punish 
bad behavior, and recognize right action. Reputation systems existed long before the 
Internet showed up and helped them spread where they are needed most — that 
space between the digital village where everyone knows everyone, and the digital 
spotlight of fame where one particular person is “known” by everyone. Online repu- 
tation systems can give everyone a kind of “fame” (or shame) that was previously 
accorded only to celebrities. 


The use of various online services to check a person’s reputation already is becoming 
universal. Consummating a business agreement, hiring a new employee, or going on 
a date with a stranger without Googling the other party is often a failure to exercise 
due diligence. Declining to participate in a reputation system or to provide some 
kind of online presence simply places one among the “unGoogleable” — which is in 
itself a kind of reputation, at least in the technology world. 
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Word of mouth, arguably the 
most ancient social control 
mechanism on the planet, is 
being redefined by the 
Internet. The important new 
phenomenon is the ability to 
aggregate word of mouth at a 
global scale and at little or no 
cost. The most immediate im- 
pact is a tremendous change 
in how companies [and peo- 


ple] market themselves. 
Chris Dellarocas 


Associate Professor 


Sloan School of Management, MIT 
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Changes to the way reputations are created, shared and revised alter 
the way markets operate and the way people interact online. Just as 
the number of online interest groups is growing, so is the number of 
online reputation systems. There are now reputation systems for 
things such as high school teachers, programmers, religious gurus, 
lawyers, journalists, resort hotel operators and charities, to name a 
few (SEE CHART, PAGE 31). 


Companies can use reputation systems to enhance customer sup- 
port while reducing its costs, and to establish trust, thereby increas- 
ing the number and quality of transactions. EBay’s feedback forum, 
which is used by millions of people for millions of transactions 
every day, is a good example. According to a study of eBay’s reputa- 
tion system by Paul Resnick, an associate professor at the University 
of Michigan’s School of Information, highly ranked sellers can 
charge about 8 percent more than sellers with no reputation, for 
identical items. 


Other companies operate online reputation systems as their core 
business, and generally follow models similar to credit-reporting 
agencies (i.e. database publishing) or ad-supported publishing. . 
And finally, some reputation systems exist to enhance the social 
agenda of their creators, such as RateMyTeachers.com or Slashdot. 


On the other hand, reputation systems aren't generally offered by 
governments, which are (or should be) constrained by accountabili- 
ty and due process. The whole point of a reputation system is to 
reflect public opinion, and people may well be judged guilty with- 
out proof. That’s fine to provide a warning, but it’s not something a 
government should be doing. For example, government sex-offend- 
er databases are not reputation systems of the kind we cover here, 
since they reflect legal outcomes rather than user-driven or peer-to- 
peer reputations. Probably the closest a government could or should 
come is to publish something such as NetScan (see pace 25), which 
offers transparency about user behavior and lets users see other 
users’ implicit reactions to participants. 
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What Makes a Reputation System 


While online forums have been in use since the 1970s, and search engines have made 


it easy to find unstructured commentary, the reputation systems we cover here deal 


in explicit data that is collected deliberately and presented consistently, usually 


quantitatively. 


Reputation systems differ in the way they address key issues of design and purpose. 


In essence, they attempt to produce objective information by managing incentives to 


participate and by putting subjective information through quality-control measures 
that act as checks and balances on individual biases. Among the factors are: 


Participants. Who’s rating whom? Is the system customer-about-buyer, or 
peer-to-peer? Do the raters have reputations themselves? Are the raters 
known or anonymous? 


Incentives. Are the participants explicitly taking part in a reputation system, 
or are they performing “normal” tasks such as writing a newspaper article or 
offering advice in a Usenet group? Does the producer of the data have an ax 
to grind? 


Criteria. What issues matter to the users? Do they care about prompt ship- 
ping or about product quality? Good spelling or accurate information? 
Bedside manner or medical outcomes? That is, what factors go into calculat- 
ing a reputation: numeric feedback from counterparts to a transaction, 
observed behavior, seals and credentials, press coverage, etc.? 


Access and recourse. Who gets to see the data, and who gets to change it? 
Who gets to know about that change? Who knows about who has rated 
whom? Can someone respond to a reputation he is assigned? Can an opin- 
ion be corroborated? 


Presentation and tools. Offline, reputation is rich and nuanced: People can 
use all five senses to determine reputation. (Is the person wearing an Armani 
suit or shabby clothes? Is the political candidate making eye contact and 
speaking clearly, or mumbling and reading from notes?) Online, users can 
only see and interact with data points. With what tools can users interact 
with and filter data? To what extent is the data abstracted or aggregated? 


23 OCTOBER 2003 RELEASE 1.0 


BOZOFILTERING IN THE WELL 


But reputation systems needn't be quite that formal. Take 
the WELL, for example, one of the longest-lived and most 
visible online communities. The WELL didn't offer an 
explicit reputation system for its users, but they came up 
with one for themselves: a feature called bozo filtering. 
Lots of services allow you to filter out the posts of people 
you don't want to read; the WELL - with a tool developed 
by users - allows users to publish their .blists (for bozo 
list). Another user tool generates the stats: who is most 
filtered, who filters most people, and so forth. The updates 


are noted, too: Who has annoyed or bored lots of people or 
suddenly regained favor? Who has lost patience and 
decided to filter all his former friends? 

Says WELL founder Stewart Brand: “I don't know 
if it's part of the WELL's success or just part of its pecu- 
liarity; it reeks of WELLness. I'm proud to be associated 
with a system that has bozo filters, and that makes them 
public.” But he adds, “The main reputation system at the 
WELL, of course, is the one | insisted on at the start: NO 
anonymity. That definitely was part of its success.” 


In the end, these choices shift the balance of power between buyers and sellers (or 
givers and takers) in complex ways. Some reputation systems are designed as explicit 


attempts at changing behavior, making teachers accountable or markets more fair 


and therefore more attractive. Buyers can evaluate sellers on multiple dimensions 
other than price. Sellers may find the payoff from a sale is extended over time by the 
permanent record in the system. Vendor-sponsored reputation systems help users 


complete transactions by enhancing confidence, steering people to good partners or 
away from bad ones. (EBay may strike some people as a slightly antiseptic place; 


indeed, it effectively marginalizes the sleazy actors.) 


This isn’t a new concept: It has been possible to see peer rankings of lawyers in the 
Martindale-Hubbell directory for decades, which tend to be sanitized by “profes- 
sional courtesy” — and the tendency of lawyers to sue. The Net, however, has made it 
easy to gather reputation data not just from peers or authorities once a year, but 


from everyone all the time, and to make the results accessible to the world. 


Context also matters. It’s easier for people to “be anywhere” online, whereas in real 
life you instinctively trust, say, the people you meet in the Admirals Club and ask 
them to guard your pc while you wash your hands or check out the fruit and crack- 
ers. (You may be wrong to do so, but that’s another matter.) In a store, you have 
some sense of whether the store has been around for a while, and you also assume 
that the clerks or security guards are keeping a watch on things. In your neighbor- 


hood, you see the same faces, even if you don’t know who they are, and you recog- 


nize strangers for what they are — interlopers with no reputation. 


A successful reputation system needs to eliminate or account for the nature of 
human judgment — and human biases. In the end, reputation system design reflects a 
social agenda and set of beliefs about human nature. Giving new power to those 


beliefs has surprising consequences. 
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Bad rap for reputation systems? 

As with any tool, the effects of reputation systems can be mixed. The value of 
increased trust and cooperation provided by reputation systems is real, but so are 
the some of the potential negative consequences of persistent, inescapable surveil- 
lance, and the continuous erosion of private space. 


The inevitable errors in reputation systems will have unfortunate consequences for 
“good” people who are mislabeled, or who find themselves dealing with “bad” peo- 
ple who have good reputations. Sometimes, those errors result from intentional mis- 
labeling: competitors out to damage each other’s reputations, con artists out to make 
themselves seem trustworthy, or mediocre workers in search of a better job. 


But other “errors” are less concrete. People may behave differently in different cir- 
cumstances and to different people; a reputation is only what some people might 
think, not a hard truth. 


Finally, good reputations are valuable. In the offline world, identities are usually 
stolen to make use of a good reputation for financial gain. It is difficult to steal a 
social rather than a financial identity in the physical world, and the practice is 
unusual enough to underlie tales such as “The Return of Martin Guerre” or “Catch 
Me If You Can.” (Assuming a new, reputationless identity is easier and much more 
common.) But online, a social reputation is easier to steal - in games or chat rooms, 
for example - and can provide emotional or psychological satisfaction to some peo- 
ple. (We leave the rest to your imagination.) 


Identity matters 

Underlying any successful reputation system must be an identity infrastructure. 
“Identity is the lynchpin of association; it’s the key that gets me unique information 
about your history,” say Microsoft’s research sociologist, Marc Smith. “You can make 
an analogy to a database, in which identity is the key or pointer, and the reputation is 
the record. Knowing your identity gets me unique information about your history.” 


Online identity exists on a spectrum ranging from completely anonymous, to persis- 
tently pseudonymous, to authenticatable identity: provable, real-world name, 
address, and other identifying data. You can also have a reputation (a credential) 
without a distinct identity, which happens more often in the real world: Everyone 
you meet in the Decathlon Club, for example, is either a member or a friend of a 
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member. Everyone at the wedding is linked to either the bride or the groom. A per- 
son holding a $100 bill is financially capable. 


The identity policies of an online reputation system affect how that system will func- 
tion and what purposes it can serve. When non-cash money or tangible goods 
change hands, both parties usually know the other’s identity or at least have some 
context. Sometimes the real names of the buyer and seller are revealed only when 
both parties have agreed to a transaction, as on eBay. Reputation systems that oper- 
ate based on the true name of all parties tend to recognize and account for positive 
contributions by particular people, which helps identify “angels.” But that same 
transparency fosters a fear of retaliation that discourages public naming of other 
shortcomings making these systems less useful for avoiding bad actors. 


Sometimes sources (or raters) are allowed to retain their privacy, but make public 
comments on identified targets, as on Travelocity, where identifiable hotels are rated 
by pseudonymous guests. In other circumstances, as on RateMyTeachers, providing 
privacy protection may be necessary to get sources to provide any data at all. Finally, 
in systems such as NetScan the reputations are derived from the behavior of mostly 
unknown or pseudonymous actors. 


SOURCE (RATER) IS KNOWN SOURCE (RATER) IS ANONYMOUS 


TARGET IS e Someone in the Decathlon e Someone holding cash 
UNKNOWN Club 


TARGET IS e Discussion groups such as e Brokered markets 
PSEUDONYMOUS NetScan e P2P networks 


e Black markets 


TARGET IS e Social accounting systems e RateMyTeachers 
KNOWN e Some merchant reputation e Shopping.com 


systems e “whistleblower” sites 


e eBay (post-transaction) 


The identity structure and policies of a reputation system affect how that system will function and what 
purposes it can serve. 
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While it’s possible to build effective reputation systems that incorporate some degree 
of anonymity for the raters, anonymity also opens the door to variety of troubles. It 
can make it harder to trust the data put into a reputation system, since it may be pro- 
vided by a malicious actor. It also makes it harder to stop bad behavior, because bad 
actors can re-appear with new identities or create multiple versions of themselves. 
“The problem is that reputation systems are intrinsically linked to identity and auth- 
entication, [which] are non-trivial, technologically, and socially,” says Slashdot cre- 
ator/director Rob Malda. “Slashdot has little way to encourage a user to retain an ID.” 


So why fool with anonymity? The short answer is that it can improve the quality and 
honesty of data collected. Reputation systems in which both parties are identifiable 
skew towards positive ratings, which may be fine for systems designed to identify 
good participants, but they may fail to surface and resolve complaints. 


Some reputation-system designers have tried to solve the inherent problems of 
anonymity technically, such as by watching for a single IP address that is originating 
multiple IDs and system entries. (Both eBay and Ratemyteachers do this). 


A more interesting approach is to adjust the overall economics of the reputation sys- 
tem to give more value to established reputations, and make them harder and more 
expensive to earn. As reputations become more valuable, users have incentive to 
remain in good standing — or at least they have to act “good” longer to get a good 
reputation. They may not bother to earn a good reputation at all, or they may dis- 
cover the benefits of being good and stay that way. (Or they may go really wild when 
they finally blow their cover and act badly.) Online games that require hours of 
patient work to create a character with a certain level of skill are an example of this. 


Makers of Markets: Driving Transactions 


The most famous reputation system of all is actually ancillary to its sponsor’s basic 
business: eBay uses its reputation system as a tool to raise users’ confidence in its ser- 
vice and to raise the actual quality of the service by driving out bad actors and 
encouraging the rest to be good. Epinions uses some of the same techniques to raise 
buyer’s confidence in the reliability of the information/reviews it publishes about 
products. In both cases, the reputation system indirectly encourages users to engage 
in transactions and increases the sponsors’ business and ultimately profits. 
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Ebay’s system is peer-to-peer, in the sense that buyers and sellers rate one another. By 
contrast, Epinions hosts a platform in which reviewers rate products and the reputa- 
tion system, which rates reviewers, raises the quality of and trust in the reviews. 
Customers have a better understanding of the products they are buying and thus 
buy with greater confidence and are more likely to get what they were expecting. 


Reputation systems work best when most people participate and contribute what 
they know, but that presents reputation system designers with a trade-off. One rea- 
son eBay achieves high participation rates is that it doesn’t ask for much: just a posi- 
tive or negative rating and maybe a comment. 


Other systems convince a smaller percentage of people to contribute more detailed 
information. Tapping the tiny minority of people willing to produce large volumes 
of good content is critical to reviews at Amazon and Yahoo! as well as Epinions. 


When those raters are the subject of a reputation system rather than paid reviewers, 
interesting things happen: Individuals compete for status by producing increasing 
numbers of high-quality reviews, as at Epinions. For example, Harriet Klausner once 
was an acquisitions librarian in Pennsylvania; today, she is Amazon’s number-one- 
ranked reviewer, based on votes from readers of her reviews. She has written 5561 
book reviews for the site — without being paid for her services. 


eBay: People are basically good 

“People are basically good” is an article of faith at eBay, printed on the back of each 
employee’s ID badge. And in a sense, the company has the data to prove it: More than 
99 percent of the transactions completed on eBay result in a positive rating for both 
buyer and seller, and only about 1 in 40,000 auctions ends in fraud. “The system 
enables people who don’t know each other to trust each other enough to exchange 
goods they can’t see until transactions are completed,” says Brian Burke, senior man- 
ager for rules & marketplace policy, who with Amjad Hanif, senior product manager, 
is primarily responsible for maintaining and improving the feedback system. 


It appears its feedback system has evolved into something surprisingly robust, foster- 
ing good behavior on a scale unprecedented in the real world. How much everyone is 
really happy, and how much the system forces “happiness,” is unclear, but it is clear 
that the eBay world is a trusting one. EBay is perhaps the online world’s best argu- 
ment for transparency. And its own reputation is such that its very name adds credi- 
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bility to “eBay inside” companies such as AuctionDrop that resell goods on eBay for 
individuals who are too busy to trade themselves. 


While the feedback system may provide peace of mind for eBay users and keep most 
people relatively honest, the primary economic beneficiary of the system is eBay 
itself. In fostering trust in the overall marketplace, not just in individual participants, 
the feedback system drives transactions and provides resistance to serious competi- 
tion either from other markets or from circles of frequent traders going offline 
(although of course that happens). The system has reduced user hesitancy about 
trading on the system by lowering the cost of verifying trustworthiness, and eliminat- 
ed the one-time aspect of the transactions. A new user may not have a relationship 
with each individual, but eBay does, and that trust is largely transferable. 


EBay’s feedback system was developed in 1996 by founder Pierre Omidyar, who now 
sits on the board of the Santa Fe Institute, an organization that studies how simple 
rules iterate to create robust, complex systems. The feedback system has 1.3 billion 
comments on file, with 2 million more being added every day. Though the basic sys- 
tem appears extraordinarily simple to users, it comprises a complex web of compet- 
ing interests and priorities resolved through a mix of technical solutions, usage 
policies and social custom. 


The two guiding principles in the feedback system’s design are reciprocity and trans- 
parency: Trading partners can rate each other; in fact, it’s expected. And it’s possible 
for anyone viewing a rating to know something about the person who provided it, to 
contact him for clarification, to read the ratings he received from others, and to 
judge whether he is likely to be a source of valid information. And of course the per- 
son rated can also see that information and can respond to his own ratings. 


The culture of reciprocity is key to getting users to provide feedback. It is also key to 
the positive-sum game: Trading partners tend to mirror the feedback they receive 
from each other, and courtesy feedback is a social norm. “When you participate [by 
rating someone] you likely will improve your own reputation at the same time, and 
that is a huge incentive,” Hanif says. As a result, more than 50 percent of transactions 
between new trading partners generate feedback. 


With a diverse base of users ranging from Fortune 500 companies such as Dell and 
Disney, to dedicated buyers of high-ticket items or distressed merchandise, to indi- 
viduals making their first transaction, one of eBay’s biggest design challenges is bal- 
ancing complexity and ease of use. 
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SQUARETRADE 


Another mechanism for establishing reputation and trust 
comes from a company called SquareTrade, which pro- 
vides a seal of approval for qualified participants who 
promise to abide by the decisions of its online dispute res- 
olution service. (DISCLOSURE: ESTHER DYSON IS AN 
INVESTOR) Users who want to improve their reputation 
online can pay for SquareTrade's seal and its services, 
which is rather like posting a bond or using a third party to 
provide some type of assurance. It's a kind of branding on 
the cheap. . .as long as you deserve it. 

Though the seal is essentially a binary rating, 
even simpler than eBay's -1, O, +1 scheme, it also carries a 
guarantee of satisfaction. Thus SquareTrade addresses the 
problem of marginally unhappy users unable or unwilling to 
post negative feedback. "Most disappointment online is on 
the fringe, not outright fraud," says Square -Trade founder 
and CEO Steve Abernethy. "Something didn't arrive on 


time, or is a little different than reported." 

SquareTrade's online dispute resolution system 
allows people reach a satisfactory resolution without con- 
sulting a human mediator by walking them through com- 
mon misunderstandings and helping them to find a 
common compromise. In effect, the system tells disap- 
pointed buyers and sellers: "Most people with this com- 
plaint resolved it this way," and offers remedies such as 
return of the item with the shipment paid, payment of a 
repair bill, or a replacement item. 

Interestingly, one of the common disputes is over 
eBay feedback (a key feature of SquareTrade's seal pro- 
gram is instant notification of negative feedback). People 
unhappy with merchants who display the seal can use 
SquareTrade before leaving negative feedback, and people 
who have received negative feedback can ask SquareTrade 
to resolve the dispute to get the feedback removed. 


The company has opted for a relatively simple system — people are rated —1, 0 or 1. 
While the summation of many ratings creates additional precision, as do the 80- 


character comments allowed with each rating, the company has resisted the tempta- 
tion to move to a ten-point scale or to allow ratings on different aspects of a trader 
or transaction, in order to keep the noise level to a minimum. 


Limiting the scope of what is collected has improved other aspects of the system as 


well. When the system launched in 1996, users could rate each other not only on 


transactions, but for any reason. Participants on message boards thus gained favor- 
able (or unfavorable) ratings based on postings that didn’t directly reflect their trad- 
ing partners’ experience. Competitors sometimes gave negative ratings to one 
another. As a result, about a year after the site launched the company decided to 
restrict ratings to completed transactions only, which Burke and Hanif say reduced 
ratings inflation in both directions. 


EBay also limited ratings to one between each pair of trading partners, rather than 


one per transaction. While this has prevented collusion among small groups of peo- 
ple intent on gaining high scores, it has been disadvantageous for those who trade 
within a limited circle and want to branch out. 


Was it good for you, too? 


As mentioned earlier, the feedback system skews positive. Again, this is partially dri- 
ven by reciprocity: Juan is unlikely to give Alice a bad rating because he doesn’t want 
a negative rating back. And because positive feedback has value in driving future 
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transactions, it is sometimes used as a negotiable item in current transactions: The 
threat of negative feedback may be used to extract additional concessions, such as 
faster shipping. Over the long run, however, these dynamics drive low-reputation 
people off the system and enhance the reputations of those who stay. (If Alice had 
such a bad reputation that her negative opinion of Juan wouldnt count, he probably 
wouldn’t trade with her in the first place.) It all fits in with the ethos that people are 
basically good. 


Yet, Hanif points out, “Feedback is a very personal experience. It is two members rat- 
ing each other, and brings all the complexities of human interaction, including the 
emotions and misunderstandings. Everyone remembers who gave them their first 
negative feedback and why.” Burke adds, “No one likes being criticized, yet the reality 
is no one is 100 percent perfect.” 


“We have a lot of big retailers that sell on the site, and their first experiences were eye- 
opening,” Hanif continues. “In retail, if you have a disgruntled customer, no one 
knows. On eBay, it’s quick for the whole community to find out what they are doing.” 


Power seduces the powerless (into being nice to the powerful) 

All this points to what is perhaps the most serious challenge: ensuring that members 
provide accurate data. Unlike most online reputation systems, feedback on eBay gets 
recorded only when auctions are completed and eBay gets paid. 


Developing a reputation costs money, and that discourages efforts SS 


to game the system just for sport. EBAY INFO 
Headquarters: San Jose, CA 
Reciprocity is surprisingly double-edged in its effects on rating Founded: September 1995 
accuracy. As mentioned earlier, it acts as a brake on unfair negative Employees: 4000+ 
ratings: Traders can “punish” what they regard as unfair negative Key metrics: $508.6 million In profits on 


oa: . . . 1.2 billion i ; $22 billi 
feedback by providing negative feedback on their trading partner. PLEDON eee teetece pian 
F A x È s in gross merchandise sales among 
This system provides an incentive for users to resolve disputes ae 
75 million registered members 


before publicly posting feedback — a service that has become the 


URL: www.ebay.com 


core business of SquareTrade. (SEE Box, PREVIOUS PAGE.) 


On the other hand, fear of retaliation and the desire for positive feedback may lead 
users to refrain from complaining even when it is warranted. In effect, users are 
sometimes forced to choose between acting in their own interest (e.g. providing pos- 
itive feedback to a marginally bad trading partner to get a positive rating in return) 
and that of the overall community (e.g. warning the community but suffering retal- 
iatory negative feedback). 
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To some extent, this reflects another tension — between users and eBay itself. For 
eBay, part of the goal is to create an atmosphere of trust. Outing the serious con 
artists is helpful to the overall system; noting every minor gripe is not. “Only one in 
200 transactions gets a negative feedback,” Resnick says. “It’s likely that more people 
are mildly unhappy, so it seems useful info is getting lost.” 


This inclination to emphasize trust manifests itself in the presentation of ratings as 
well. “EBay does not let you search or sort through feedback ratings,” says Brian 
Dear, the founding director of eBay Design Labs and now working on a startup still 
in stealth mode. “All you can do is page through the feedback ratings sequentially. 
One consequence: after slogging through page after page of positive ratings looking 
for those curious negatives, all that positivity erodes your desire to keep looking.” 


Not all the efforts to maintain data integrity are based within the feedback forum. In 
particular, eBay’s identity system, which contains full names, addresses, and credit 
information, is key to fraud detection and prevention. EBay watches for people who 
are rating themselves and for collusion among groups by monitoring incoming IP 
connections and suspicious patterns of buying and bidding behavior such as sudden 
increases in average transaction values. 


It’s worth noting that Alice’s perceived reputation is also based on other public 
information, such as how long she has been on the system, what kind of good she is 
selling, how she presents the items she has for sale, and how she responds to ques- 
tions. Of course, these inexplicit cues that people use to form opinions about a per- 
son’s reputation aren’t always reliable either: Serial murderer Ted Bundy used his 
supposed good looks to gain the trust of his victims. 


Epinions: A web of trust 

Founded in May of 1999 by Nirav Tolia and four others from Yahoo!, Netscape and 
Excite@Home, and launched with $8 million in funding from Benchmark Capital 
and August Capital, Epinions was one of the early efforts to aggregate word of 
mouth among shoppers on the Internet (SEE RELEASE 1.0, MARCH 2001). 


“We took our inspiration from Amazon for its user experience and user-generated 


content, and from eBay as the best marketplace,” Tolia says. “We wanted people to 
rate the reviews and the reviewers, and to build a marketplace for content.” 
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The company received $45 million in VC funding before being acquired in March 
2003 by Shopping.com (formerly DealTime), a New York-based price-comparison 
service. “Shopping.com offered pricing information, Epinions specialized in reviews, 
and people want both,” Tolia says. “Shopping.com is about where to buy things; 


» 


Epinions is about what to buy [and whom to trust] 


Users trust the information on the system because products, reviews and reviewers 
are all rated, Tolia says. Some users write reviews of the products. They and other 
users can then choose to add reviewers to their personal Web of Trust by reading 
their reviews, checking out their profile pages and evaluating the reviewers’ Webs of 
Trust. The more people who trust a reviewer (and, in turn, the more 


people who trust the person who trusts a reviewer) the greater his 


EPINIONS INFO 


trust ranking and the more likely he will be chosen by Epinions as 
an "Advisor." "We built a huge transitive web of trust that gives users 


access to the work a lot of people have done," Tolia says. 
Employees: 35 


Headquarters: Brisbane, CA 


Founded: May 1999 


Funding: owned by Shopping.com 


For Shopping.com, the merger with Epinions has helped the com- Key metric: $8 million in revenues in 
pany provide customers with unbiased information they need to 2002 
make buying decisions about what, not just where, and, says URL: wwiepinions.com 


Epinions founder Nirav Tolia, to “humanize” what is otherwise a 
commodity service. “It’s the only real differntiator,” he says. “Before 
the merger, we considered syndicating our content, but we realized we could bring 
more value to a single partner.” Shopping.com powers other shopping sites such as 
Earthlink and Excite, but it syndicates its merchant relationships and Epinions con- 
tent only as part of a total package. 


Tolia suggests merchants such as Sony could benefit by referencing Epinions reviews 
as well. “Using Epinions reviews won't muddle their brand, and their customers 
won't feel like ‘these guys are trying to sell me,” he says. And because disputes are not 
always the fault of the merchant, Epinions provides the benefit of a feedback mecha- 
nism to allow sellers to challenge negative reviews. 


Today, the company collects about 500 reviews of products, merchants and reviewers 
from 300,000 to 400,000 unique visitors per day, and more than 1 million product 
reviews are available on the site. Total transaction volume in 2002 at Shopping.com 
topped $1 billion, making it the fourth-largest shopping site on the Net, after eBay, 
Amazon, and Yahoo!. Of the $38 million in pro-forma revenues generated by 
Shopping.com, $8 million is directly attributable to Epinions, says Tolia. Nearly all of 
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this revenue is paid by merchants on a cost-per-click basis, though the company has 
a few revenue-sharing arrangements. 


To manage the flow of reviews, three Epinions employees work with 32 independent 
and mostly unpaid “category leaders,” who in turn help supervise 300 to 400 com- 
munity leaders known as top reviewers or advisors. Category leaders don’t necessari- 
ly write lots of reviews themselves; their role is to encourage reviewers/advisors and 
to ensure overall quality. 


“You have to make a self-organizing system, and the keys to that are structure and 
feedback,” Tolia says. “People need a structure so they know what you expect from 
them, and then you need feedback because the only way to manage that much out- 
put is to have it manage itself” 


Key to the system’s effectiveness is providing incentives to visitors to write high-qual- 
ity, but not necessarily positive, reviews. Initially, the company paid writers 30 cents 
per page view, which was not only expensive but encouraged quantity over quality. 


In 2002, financial considerations forced the company to adopt a new payment struc- 
ture for reviewers. The company uses a complex algorithm to calculate a dollar value 


IC 


for users’ “shopping” and ratings in a particular category — i.e. reading several 
reviews on a given model of toaster oven, setting up comparison panels to compare 


like models, etc. That amount is then split among reviewers in that category. 


While the payment system bears similarities to pooled tips for waitstaff in a restau- 
rant, the incentive to provide good “service” does not: Epinions discovered that 
many of the best reviewers werent doing it for the money. “People were getting 
intrinsic satisfaction out of sharing their opinions and getting feedback,” says Alexis 
Johnson, Epinions’ community product manager. “Many of our reviewers indicate 
they aren’t writing for the money by donating their earnings straight to charity.” 


The new payment system also encourages both positive and negative reviews. “We 
want to make sure we are incenting people to be impartial,” Johnson says. “Our read- 
ers love negative reviews. Frequently, they will search by star ratings and read [only] 
the one-star items.” This makes sense: After filtering out the lemons, users make 
choices based on price, features and availability. 


Interestingly, while the reviewers are willing to give thumbs up or thumbs down on 
products, they are highly sensitive to negative feedback themselves. “There’s an 


WWW.EDVENTURE.COM 


unwritten rule: Never give negative feedback without explaining 
why,” Johnson says. “There are people who hate each other over that 
kind of thing.” 


And this leads to a potential weakness in the system. The reputation 
of reviewers doesn’t necessarily map to the quality of reviews. 
“When we first started there was this belief if you allowed people to 
trust each other, they would be rational about whom they trust and 
why. The people with the best ideas and writing would have the 
most people who trusted them and would filter to the top. But that 
assumption wasn’t correct,” Johnson says. 


“Sometimes, the one who has gained most trust is the guy who has 
asked people to trust him, or is the funniest or the sweetest,” 
Johnson says. “You can also have great writers who don't play the 
community game: They want to write reviews and leave, and since 
they are not actively involved they don’t get noticed and trusted.” 


For now, Shopping.com is living with this, and considers it a reflec- 
tion of the way things work in the rest of the world: Influence does- 
n't always map to competence. 


At the same time, the company is careful to avoid cheaters. It relies 
on its army of reviewers and editors and the sheer accumulation of 
commentary to prevent manufacturers, merchants and reviewers 
from gaming the system. “Unless you are the manufacturer of the 
product, there is very little reason to cheat, but we have some auto- 
mated things, like IP logging and cookies, that help us identify peo- 
ple who are submitting false reviews,” Johnson says. 


Imitation is the sincerest form of reputation enhancement 

For other vendors thinking about building their own reputation- 
based rating systems, Johnson emphasizes the importance of 
Epinion’s early decision to pursue writers who had posted high- 
quality reviews on Usenet and other sites. “We launched with 2000 
high-quality reviewers. Handpicking the people to start the com- 
munity was really important,’ Johnson says. “You have to choose 
the leaders before you open the doors.” Call Usenet and other sites 
farm teams for the commercial enterprises. 
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Reputation, like identity, is 
by its nature ambiguous, and 
that is a feature, not a bug. 
Computers are good at 
drawing sharp but not fuzzy 
distinctions; explicitly defin- 
ing who is a friend can kill 


relationships. 


Cory Doctorow 
Outreach Coordinator 


Electronic Frontier Foundation 
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It’s also necessary to find ways to separate users into subgroups. Within Epinions, 
three relationships are possible: Users can trust a particular reviewer, ignore him, or 
put him in a killfile so they won’t even see his comments or reviews. The default is to 
see only reviewers that have not been blocked by much of community. If a user reg- 
isters and signs in, he can set his own filters and preferences. These filters allow dif- 
ferent and diverse communities to co-exist within the site: If you don’t want to see 
certain people, you don’t have to. 


Watch Out: They're Watching You 


Another business model for reputation systems is simply publishing the informa- 
tion, with revenues generated either directly from the users of the information or 
from advertising. These reputation systems aren’t peer-to-peer, but rather focus on 
collecting reports from one community about individuals or organizations in 
another community — buyers and sellers, typically. 


Most individuals already have a financial reputation lodged in the databases of 
Experian, Equifax and TransUnion. As with other reputation systems, these services 
function in part to establish trust and reduce risk by providing predictive informa- 
tion about a buyer’s or borrower’s ability to pay. For consumers, they have changed 
everything from how mortgages are issued, to how automobiles are leased, to how 
home rental applications are evaluated. Similar systems from Equifax and Dun & 
Bradstreet exist to evaluate the credit risk of companies: Do they pay on time? How 
big are they? Based on that information, sellers can decide what terms to offer. 


Credit is much more available in economies with such systems than in newer “free” 
markets such as those in Eastern Europe, where companies are loath to extend credit 
to unknown consumers or small businesses. 


But until recently, these systems didn't do much to influence user behavior. Since the 
information was available only to paying customers, the individuals and companies 
rated often didn't even know what their ratings were and couldn't adjust their behav- 
ior in response. Now, consumers can get easier access to their own ratings, the result 
both of legislation and of the the distribution economies of the Web. 


Internet-based reputation systems confront users with their reputation, which causes 
people to adjust their practices. That's been true at OpenRatings, which turn the old 
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credit-scoring model around by gathering the same sort of data about suppliers, and 
selling it to the suppliers’ customers (as well as to the suppliers about themselves). 


And finally, new tools such as Overture and Google AdSense make it easier to fund 
reputation systems through advertising by matching the market and the advertisers. 
Information-sharing sites such as Slashdot take advantage of this model. 


Slashdot: Working for Whuffie 

Awarding recognition is a powerful way to inspire individual contribution to group 
projects, and the desire to astonish friends has always been a powerful force in any 
creative community. That’s particularly true in the technology-developers’ commu- 
nity, which in large part is driven by “Egoboo” (a term popularized by Eric Ray- 
mond), “Whuffie” (popularized by Cory Doctorow) or “Karma” (from Slashdot 
itself), all terms used to describe the combination of psychic satisfaction and social 
credit, or “reputation capital,” that result from building something of value and 
being recognized by peers. 


Several sites have built systems to track reputation in the technical community. By 
far the biggest and busiest of these is Slashdot, a website on which members can post 
and comment on stories and developments of mutual interest. Participants gain 
Karma, or reputations, based on the quality of their contribution to discussions. 


Slashdot’s Karma is measured as Terrible, Bad, Neutral, Positive, Good or Excellent. 
It is used to identify users who abuse the system, to distribute the work of moderat- 
ing discussions, and to float “good” postings to the top. “We use it as a defensive 
mechanism to prevent flooding, denial of service and patterned abuse,” says Slashdot 
creator/director Rob Malda. “Our fundamental goal is to allow readers to decide 
how much time they want to spend on any given story. If a reader wants to see only 
five comments, we want to make sure they get five decent comments.” 


Users with good Karma can speak a little louder. They may also be awarded modera- 
tor access, which allows them to judge a limited number of posts by other users. 
Finally, moderators themselves are judged by “meta moderators,” groups of five 
(randomly chosen, Karmically qualified) users. 


The function and purpose of the reputation system have evolved over time. “I didn’t 


have anything lofty planned at the start. I just needed to clean up the forums when 
trolls flooded them and interfered with my ability to read the site,” Malda says. 
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Slashdot’s users, however, have come to equate Karma with prestige, and that makes 
all the difference in how they behave. Malda notes that ratings were initially known 
as total mods, and largely ignored until he changed the name to Karma, mostly 
because it was humorous: “It was my lesson in the importance of naming: 
[Suddenly] people [were] abusing Karma left and right, gaming the 
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system to the point where they had no accountability.” 


Headquarters: Ann Arbor, MI 
Founded: September 1997 


Employees: 10 


Funding: acquired by the Open Source 


Of course, this particular community is partially defined by its 
members’ love of breaking stuff to see how it works. As a result, 
Malda is looking at several approaches to limiting the impact of 


Development Network hackers cracking. “I want the system to have ever better memory of 
Key metric: ~ 10,000 posts and what has happened historically, so that the effects of moderation can 
BOR POR readers per day be scaled depending on all available data,” Malda says. 


URL: www.slashdot.org 
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In particular, Malda is exploring ways to derive ratings from techni- 
cal data, somewhat like NetScan (see pace 25). For instance, a post 
from an IP address from a subnet with a history of bad posts might be given a nega- 
tive rating by default, rather than the normal positive rating for posts by registered 
users. “Right now we make little use of tons of data that could help us such as time 
span between post and moderation, the user’s posting history, and the subnet post- 
ing history,” he says. “There are probably 100 subnets with 5 percent of our traffic 
and 90 percent of our crap.” 


Slashdot is also operating a web of trust system, called the Zoo, that lets users identi- 
fy friends, foes, friends of friends, and foes of friends. Users can trust input from 
friends and friends of friends, and block data from users they don’t respect. “We 
don’t yet use the data for much beyond novelty, and simple things like killfiles, but I 
would like to see a true [vouching] system where I stake my [reputation] against the 
future posts of another person,” Malda says. 


RateMyTeachers.com: Who grades whom? 

Educational reformers have argued for years that merit pay (based on accurate 
assessment) could improve school performance. Though teachers routinely grade 
their students, it’s only recently that large-scale, public systems for grading teachers 
and professors have come online. 


The anonymous nature of many of the participants in this system makes it harder to 
trust their results, but for those of you with kids in middle school or high school, we 
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suggest you ask them how well RateMyTeachers.com performs. (Other readers may 
want to check out (or perhaps create) the hypothetical RateMyBoss.com or even 
RateMyVC.com.) 


“Students are always talking about their teachers, but it has never been organized 
and collected like this before,” says Michael Hussey, co-founder of RateMyTeachers, 
which hosts nearly 3 million mostly anonymous reviews of more than 400,000 
teachers in more than 24,000 schools and school districts in the US and Canada. 
Previously, Hussey founded infinite MEDIUM, which created a number of other rat- 
ings sites including RateMyPet, RateMyWheels and RateMyRecipe. He developed his 
first rating site, RateMyFace, when he was studying financial economics at the 
University of Maine in 1999. 


Currently self-funded and supported by ad revenue, the company was founded in 
April 2001 as a partnership between Hussey, MisterMessage (which developed other 
RateMyxX sites, but now focuses on RateMyTeachers), and John Swapcienski, the 
owner of RateMyProfessors, a similar service focused on university professors. 


“We want to improve education, and that means recognizing the bad teachers and 
praising the top teachers,” says Hussey. But clearly, there is more to it than that. To 
the extent these services function as whistle-blowing systems, they must offer public 
anonymity for reviewers in order to be effective. Yet that presents its own set of 
problems. While proponents see these systems as a method of safely speaking truth 
to (or about) power, anonymity also provides a haven for spiteful, 


vengeful underperformers to unfairly tarnish the reputations of 


teachers they don’t like. 


Hussey has received several threats of lawsuits, and more than 150 
schools and school districts have restricted access to the site. In 
Winnipeg, Manitoba, education officials made a concerted effort to 
track down and punish students who had posted reviews. 


Given the anonymous nature of the postings, ensuring the quality 
and fairness of the reviews on the site is a challenge. Hussey notes 
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Headquarters: Bakersfield, CA 

Founded: April 2001 

Employees: 7 

Funding: self-funded for about $35,000 

Key metric: 400,000 teachers rated in 
24,000 schools 


URL: www.ratemyteachers.com 


that more that 1600 volunteers are engaged in checking the 10,000 reviews that are 


added to the site every day, and that the company is actively recruiting “administra- 


tors” to monitor the postings about teachers at every school in its database. On the 


back end, the site watches for multiple entries from the same source or IP address. 


23 OCTOBER 2003 


RELEASE 1.0 21 


Hussey adds that a number of projects are under way to improve the system; among 
them is a feature that will allow students to rank the reviews posted to the site. 


“Our main focus is to discover if teachers are respected and if students in their class- 
es are learning or bored or frustrated, and we are doing a good job of that,” Hussey 
says. Even with anonymous reviews, he adds, “you can tell who is respected in their 
class and who is effective as a teacher.” 


While in some ways the site functions to provide sanctions against abusive teachers, it 
turns out that 60 to 80 percent of the reviews are positive, and that some of the most 
negative commentary comes from teachers talking about their colleagues. Hussey 
notes that the objections from some are balanced by favorable commentary from 
others who say the site has helped them to improve their classroom effectiveness. 


Take Dennis Buonafede, an Ontario, Canada-based high school teacher who asked 
his students to use RateMyTeachers to help him improve his teaching. “As a new 
teacher, I’ve often wondered what my students thought of me and my teaching abili- 
ty,” he says. Understandably, the majority are either too shy or too concerned with 
possible repercussions to speak out in class. “I ask them to be fair and respectful of 
my feelings, but not to hold back on what they think.” Once there was enough of a 
representative sample, certain patterns developed. “Admittedly, some of their com- 
ments stung, others were (I think) unfair, and many were encouraging,’ Buonafede 
continues. “Whether good or bad, they all helped me improve.” 


Currently, RateMyTeachers is eking out a profitable existence based on ad revenues, 
largely provided by Google AdSense. “We feel we offer an interesting and valuable 
opportunity to reach a significant localized audience of savvy teenagers,” Hussey 
says. With usage growing roughly 10 percent per month and doubling every six 
months, those ad revenues may amount to something worthwhile over time. 


OpenRatings: Source control 

Sellers have long had access to the credit histories — i.e. the financial reputations — of 
most of their customers. But buyers don’t generally have the same level of informa- 
tion about suppliers. 


“Sellers invest a lot in generating a favorable reputation for themselves,” says 


OpenRatings co-founder and CEO Stan Smith, formerly vice president and general 
manager of Agile Software, a maker of content management software. “They make 
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sure you only find out what they want you to know, but the buyer’s creditworthiness 
is exposed by a third party” such as Dun & Bradstreet or Equifax. 


In turn, this gives some unfair economic and informational advantages to sellers. 
“Buyers have been paying [both] for their own credit risk as perceived by the seller, 
and for the performance risk of the seller,” Smith says. In other words, when sellers 
fail to deliver, buyers have little recourse, and no way to share that information with 
others. Are the sellers meeting their delivery dates? Are they having quality prob- 
lems? Cash flow problems? 


OpenRatings’ reputation system helps buyers in complex supply chains predict 
which of their suppliers may perform badly. Begun as a project at MIT’s Media Lab 
by company CTO Giorgos Zacharia while a student there, the company counts 
United Technologies, Xerox, United Airlines and Rockwell Automation as customers. 


Dun and Bradstreet, an investor in OpenRatings, provides much of the data used to 
track and analyze more than 40 million suppliers. OpenRatings also gathers infor- 
mation about suppliers directly from customers, including data 

about on-time delivery, quality, cost, customer support, personnel 


changes, responsiveness, order accuracy, reliability and overall per- OPENRATINGS INFO 


formance against competitors. 


Headquarters: Waltham, MA 
Founded: May 1999 


Finding and interpreting correlations within its own data and that Employees: 27 
from D&B is OpenRatings’ mission. “There are a number of things Funding: $25 million from Ampersand 
companies do to respond rapidly to problems in their supply chain Ventures, Atlas Ventures and Dun & 


but we are about prevention rather than finding a cure,’ Smith says. Pra eSHpeL 


“Part of our intellectual property is [knowing] what patterns in 


which industries with what-size companies mean something.” 


Key metric: profitable 


URL: www.openratings.com 


Some of the correlations are counterintuitive. For example, compa- 

nies that are just getting into trouble often pay off existing debts early to clean up 
their credit scores. Then, they either obtain additional credit, or they fail. Of course, 
the early warning provided by a cleaner credit score works only when that informa- 
tion is compared against other data gathered on the company. 


To ensure accuracy (and reduce rancor), OpenRatings offers suppliers the ability to 
see, challenge, and correct ratings given by customers. In practice, however, Smith 
says that digging deeper into the reasons for poor ratings usually reveals additional 
problems rather than inaccuracies. 
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A growing number of suppliers use the information in the OpenRatings database to 
monitor their own performance with a specific customer or overall performance. 
“We even have companies that use their performance feedback/scores as a basis for 
their executive compensation; it’s real pay-for-performance,” Smith says. Suppliers 
also use OpenRatings to provide information on past performance to the General 
Services Administration — a requirement for suppliers bidding on federal contracts. 


Surprisingly, the clarity the system offers tends to repair relationships rather than 
destroy them, even with troubled companies, says Smith. In the world of supply 
chains, trading partners tend to have long relationships over many transactions — a 
sharp contrast to the large numbers of one-time transactions between strangers at 
eBay. Finding a new supplier is expensive, and in most industries, large companies 
change only 5 to 15 percent of their suppliers per year. “[In the past], suppliers have 
tried to hide their problems until it’s too late,” Smith says. “Instead, we have cus- 
tomers call their suppliers and say ‘we know you're going to be having a quality 
problem, and we want to work with you on that” 


Sleeping well at Rockwell 

Rockwell Automation, a 22,000-person company based in Milwaukee, WI, has been 
using OpenRatings for about six months to monitor the financial health of roughly 
3000 suppliers. “We really wanted a more disciplined approach to supplier selection 
and risk management,” said Mike Sparger, director of purchasing for Rockwell’s 
Control Systems Group. “It has already our changed relations with our suppliers. 
When the OpenRatings system sends us e-mail saying “You have a supplier that is in 
trouble, we get in touch with them.” 


Sparger says Rockwell has contacted several critical suppliers about negative infor- 
mation and has moved to reduce its risk by stockpiling inventory, asking suppliers to 
build ahead and looking for second sources. The company has also used Open- 
Ratings’ supplier-assessment tool to help troubled suppliers improve their processes. 


The system has also changed the supplier-selection process. “We also use it in the 
front end of the process.We run the OR data on every candidate,” Sparger says. 


Currently, most of data provided by OpenRatings is based on information from Dun 
& Bradstreet and is supplemented by data provided by other OpenRatings cus- 
tomers. As a relatively new user, Rockwell is not yet feeding data back to Open- 
Ratings, but it plans to. 
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Reputation Refineries 


Microsoft's NetScan: Speech acts 

Even in conversational spaces, it’s possible to build reputation systems based not on 
what people say, but how they behave — how often they communicate, whom they 
communicate with, how they respond, and where they participate across a range of 
different discussions. This can be particularly useful in customer-relationship man- 
agement and technical support. 


One of the biggest public conversational spaces is Usenet. Though it has suffered 
from spam and disinterest on the part of many of its early users, Usenet is still used 
daily by millions of individuals. Google’s decision (in February of 2001) to offer 
access to and hosting for Usenet may also give the service new life, but Microsoft 
seems to be ahead in analyzing the space. 


Can you tell whether someone is a useful source of information about a topic, based 
not on the content of his writings, but on others’ reactions to his posts? That’s the 
possibility addressed by Microsoft’s NetScan service, which helps Usenet users iden- 
tify valuable sources of information and important topics within each group, and 
track the participation of favorite authors within and across different groups. 


Unlike explicit feedback systems such as eBay or Epinions, NetScan helps users eval- 
uate for themselves the quality of Usenet groups, as well as authors and conversation 
threads within each group, by analyzing and displaying patterns of user interaction. 
“Reputation systems don’t make angels of us all, but they make visible those who 
are, and might tempt people to be angelic,” says research sociologist Marc Smith, 
who has been working on the NetScan project since 1994. Smith started it as an aca- 
demic project at UCLA and brought it with him when he joined Microsoft in 1998. 


Today, Smith leads a team of 11 people in Microsoft Research’s Community 
Technologies Group, and NetScan serves up half a terabyte of data to tens of thou- 
sands of users each month. NetScan covers over 110,000 Usenet groups, tracks mil- 
lions of authors, and provides several different visualizations of groups, usage 
patterns and author activity. 


While Google looks at the content of Usenet messages to help users find answers to 
questions, NetScan examines Usenet message headers and conversational threads to 
identify good sources of information. “We began by asking, “What are the salient 
sociological properties latent in the data but not manifested or visible in the inter- 
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face?” says Smith, who earned his PhD in sociology at UCLA under Peter Kollock, 
an expert in “reputation studies” (SEE RELEASE 1.0, APRIL 1996). 


“Authors vary significantly in their patterns of contribution and interaction,’ Smith 
says. “NetScan makes these patterns visible so that users can make inferences about 
which patterns seem useful and which do not.” For example, systematic observation 
has shown that an author who posts 20 replies to 20 different threads is more likely 
to be providing useful information than someone who initiates one or two threads 
but never replies, or someone who posts 20 replies within a single thread. 


Valuable threads can often be identified by their structure: Who made the initial 
posting, how many replies it received, how many different authors were involved, 
the length of time between the first and final post, the number of child messages for 
each possible parent message, and so on, all provide clues. Using behavior to define 
the probable value of content allows NetScan to steer clear of subjective judgments. 
“T don’t have to say someone is a flamer; I can just say they have a low thread-to- 
post ratio,” Smith says. 


Smith characterizes the behavior of Usenet authors using four broad categories: 
answer person, questioner, debater and bursty debater — someone who comes into a 
group, posts a lot in a few threads for a short time and then leaves. 


Answerers make up only 2 percent of the population, but contribute nearly all the 
information of value in a group, says Smith. 


Usenet newsgroups can be evaluated with similar metrics. Those with a high per- 
centage of returnees and a core of active authors are more likely to contain good 
answers to questions than groups with a high percentage of one-time visitors (cf. the 
tourist restaurants). Analyzing cross-postings makes it possible to see how different 
groups are related, and which maintain topic focus. 


NetScan is developing number of approaches to visualizing interaction within news- 
groups using standard approaches to information visualization (SEE RELEASE 1.0, 
SEPTEMBER 2002). These visualizations allow users to quickly discover relevant groups 
and to assess the behavior of individual contributors within those groups. 


Sponsored but not controlled 


NetScan is currently a free service, but Microsoft is considering revenue opportuni- 
ties by providing it as an underlying platform for enterprises or community hosts, as 


WWW.EDVENTURE.COM 


it does with Exchange, or as part of a service, such as a richer ver- 
sion of MSN Communities. I don't have to say someone 

is a flamer; | can just say 
Even though it generates no direct revenue as yet, NetScan has a they have a low thread-to- 
more subtle value to Microsoft as a means of enhancing the quality post ratio. 


and reducing the cost of technical support for its products. ; 
Marc Smith 


. . g g Research Sociologist, NetScan 

While NetScan covers all of Usenet, its focused is on Microsoft g ; 
ee Microsoft 

products. The company spends more than $300 million a year on 

technical support, and that amount is growing. Says Smith: “The 

cost of a person on the phone providing support is astronomical. It 

costs dozens of dollars to pick up the phone, and that’s not because 

we are doing it wrong. That’s just the cost of keeping a trained per- 


son in a chair.” 


And there is strong interest in peer support; in August 2003, nearly 
10 percent, or 214, of Usenet groups discussing Microsoft products 
had more than 1000 users. 


NetScan also has the benefit of being Microsoft-sponsored yet 
influencing people’s attention in a way that could not be described 
as censorship: It is transparent in how it works, relies on the com- 
munity’s own content and does not modify that underlying, inde- 
pendently posted content. Such a service is a competitive advantage 
because “people recognize that you have cultivated a healthy garden 
around your company, and that just moving towards your company 
can put them in touch with helpful people,” says Smith. 


The service also includes filters that can be applied by each user, not 
by Microsoft. “Spam shows we can’t depend on technology or legal 
frameworks that assume most people are good,” Smith says. “Online 
environments attract people with obsessive-compulsive disorder, 
and give disproportionate power to that minority: .0001 percent 
can screw it up for everyone else.” 


Awarding recognition to the positive contributors, however, can 
help turn lurkers into posters, and average posters into contribu- 
tors. “If you put a hierarchy ladder in any group, some fraction will 
try to climb it, and social accounting mechanisms [which make 
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explicit the level of contribution made by participants in a system] create that kind 
of hierarchy,’ Smith says. 


In addition to fostering the growth of support communities, Smith is exploring sev- 
eral new directions. A likely improvement is hardening the identity scheme used by 
NetScan. “There should be a community server from Microsoft,” Smith says. 
“Microsoft is the company that can make it possible for 143 million people to have 
access to cheap, reliable social accounting services.” 


The need to differentiate between useful and useless participants in online conversa- 
tional spaces is also apparent in chat rooms. Microsoft’s recent decision to shut pub- 
lic chat rooms might have been different had it been possible to keep the focus on 
the valuable contributors, inexpensively. 


In the meantime, Smith is looking to apply NetScan’s technology to the 1500 e-mail 
discussion lists within Microsoft. “The dominant application of this technology may 
be inside the firewall,” Smith says. 


Yet another possibility is making a personal version of NetScan that could help peo- 
ple manage their e-mail. For example, correspondents who always get a reply within 
six hours may be judged to be important, and so their messages (and messages that 
include them on the distribution list) can be inferred to be important as well. 
(Cataphora also analyzes communications patterns to discover social relationships. 
(SEE RELEASE 1.0, MARCH 2003)) 


Biz360: PR mirror 

Biz360 condenses ephemeral commentary into tabular reputation data by analyzing 
press coverage of its customers and their competitors, keeping track of who said 
what and why. 


“We provide automated, real-time, continuous feedback that lets you measure the 
outcomes of your messaging, and ultimately, how that affects your market share,” 
says CEO Bud Michael. “We allow you to identify new media opportunities, who is 
giving you favorable or unfavorable coverage, and key trend stories that should 
include coverage of your company.” 


The company’s roots are telling. Founders You Mon Tsang (now chief marketing 
officer) and Dan Gartung (chief technology officer) were formerly with Brio 
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Software, a developer of software that helps companies measure their performance 
in areas such as sales and customer relations. Biz360 applies similar analytics to 
information outside a company. “We asked “How do you take what the world has 
learned about business intelligence inside the company to the world outside the 
company?” says You Mon Tsang. 


Its Market360 software applies text analysis and pattern recognition algorithms to 
more than 50,000 sources of content, including print, broadcast, the Web and mes- 
sage boards. It then calculates an assumed dollar value of press coverage received, 
taking into account ad equivalence, the size of the audience reached 

by the various news outlets, and the “mention momentum,” i.e. how 


often the customer receives coverage over different time periods So 


compared to its competitors. 
Headquarters: San Mateo, CA 
Founded: February 2000 


Also, if Hewlett-Packard were to analyze its recent coverage, Market 
Employees: 55 


360 would offer comparisons to the amount and tone (positive or 


negative) of coverage received by Dell, Gateway, Apple, or other Capital, Granite Ventures, BA 
selected competitors. Venture Partners and Adobe 
Ventures 


Of course, press clipping and counting services have been around 
$20,000-50,000+ per year 


for years. What’s new is the ability to condense nuance into mean- 
URL: www.biz360.com 


ingful numbers, and to help answer questions about why coverage 


Funding: $20+ million from Foundation 


Key metric: About 70 customers paying 


might be trending positive or negative. 


By manually evaluating 30 to 60 articles, customers train the software to distinguish 
between positive, negative and neutral coverage. It is also possible to track “senti- 
ment,” not just about the company as a whole, but also regarding particular mes- 
sages and campaigns. 


Other indicators ranked and tracked by the system include prominence within par- 
ticular articles, mindshare relative to competitors, “zero mindshare” publications 
that have not yet written about a company or topic, favorable authors, and various 
breakdowns by region and publication type. 


“The biggest trend here is that we are enabling corporate communications to take on 
a greater strategic role in the company,” says Michael. “Historically, senior manage- 
ment has seen PR as event-driven: The [Exxon] Valdez runs aground and spills oil, 
and you have to react.” He continues, “[Customer] Bank of America got a new CEO 
who said its goal was not to be one of the most admired banks, but to be the most 
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admired bank.” Using the Market360 software, Bank of America will be able to see 
whether it is moving toward that goal. 


Customers can also assess the effectiveness of their spending on PR and other com- 
munications activities against other options such as advertising and direct market- 
ing. Another Biz360 customer, McData, recently launched a product with no 
advertising at all, yet managed to outperform its competitors in both the press and 
the market, says Michael. 


Customers can also find new outlets for coverage by tracking reporters who are fol- 
lowing competitors and trends of interest to the company, and to time announce- 
ments more effectively in response to market developments. 


VeriSign, for example, has been using the system for about two years, and Tom 
Galvin, the VP of public affairs, says he checks the system 6 to 12 times per day. 
“Biz360 helps us around the nuances of what is going on,” he explains. “It’s invalu- 
able to see how we are being talked about.” 


Of course, Biz360 can report only what others are writing. As the recent SiteFinder 
debacle shows, VeriSign’s reputation is ultimately in its own hands. 


The company charges customers $20,000 to more than $50,000 per year for its ser- 
vices, and Michael says that revenues are doubling every year. The company has 
more than 70 customers, including 3Com, Bank of America, Cap Gemini Ernst & 
Young, Harley-Davidson, Hyperion, PacifiCare and VeriSign. Biz360 also recently 
announced a partnership with Factiva to expand the scope of its press coverage. 


The Wrap on Reputation 


Remember those ads for Dial deodorant soap: "I use Dial. Don't you wish everyone 
did?" You could say the same about reputation systems. 


They seem to benefit everyone: They foster transparency, help customers get what 
they want, increase business for vendors and assign credit where it is due. Even inde- 
pendent reputation systems, designed to foster clean markets for the public rather 
than for particular vendors, can now be self-sustaining or even profitable through 
targeted advertising mechanisms that were not available even a few years ago, and 
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NAME WHAT IT IS COMMENTS OUR RATING 


AllExperts Thousands offer free advice Worth more than you pay for it la 


www.allexperts.com 


BizRate Customers rate businesses See the tension between the FFER 


www.bizrate.com urges to sell and telling the truth 


ChoiceTrust Background checks on doctors, Filtering out lemons FEE 


www.choicetrust.com nannies, contractors 


CitySearch Reviews of restaurants, bars, What's good in the hood see 


www.citysearch.com shops, clubs, etc. 


Hot or Not Are you a hottie, or do you just Get the superficial part of dating EREE 


www.hotornot.com think you are? over with quickly 


Landlord Protection Agency Landlords rate tenants; other Non-slumlords check to see if *k 
http://www.thelpa.com/Ipa/ sites let tenants rate landlords you're a slummer 


ntrb_description 


Sarlo's Guru Rating Service Which of 1250 religious gurus is Not much feedback from the aad 
www.globalserve.net/~sarlo/ right for you? enlightened ones, unfortunately 
Ratings.htm 

The Erotic Review Customers rate sex workers Not safe for work kkk 


www.theeroticreview.com 


Zagat Survey Ratings for tens of thousands of Zagat-rated means “+1" by default wae 


www.zagat.com restaurants around the world 


How They Stack Up: Some Other Ratings and Reputation Sites: The line between ratings, comments, gossip and reputation isn't 
always so well defined. Following are some sites that gather and present a mix of structured and unstructured data to enable users to 
evaluate information or conduct business. Some are examples of services that have been around for years; others are "only on the Net." 
Most are based on first-hand experience, and all of them are suggestive of applications for reputation systems. See page 33 for more! 
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their audiences are growing as more people come online and get familiar with the 
concept. In fact, reputation systems might even foster trust where governments fail 
to do so. Only the bad guys could protest. 


So, given the wonderful and visible example of eBay, why aren't they more prevalent? 


The reasons are varied, but as we have illustrated in this issue, reputation systems are 
not magic. In addition to carefully honed rules, they require people to make judg- 
ment calls. A reputation system needs to be seeded properly and weeded over time. 
Also, eBay works closely with law enforcement when problems arise, so eBay's sys- 
tem would not necessarily work as well without a credible government in the back- 


ground. At the same time, reputation systems are not bound by the constraints of 


COMING SOON 


e Social network services. 


e Nonstructured data analysis. 


e Maps and geolocation. 
e The future of e-mail. 
e State of the Net: Is our infra- 


structure sound? 


e And much more... (If you 
know of any good examples of 
the categories listed above, 
please let us know.) 


due process that mostly keep governments from exercising arbitrary 
power precisely because reputation systems do not have statutory 
authority over anything. All they can do is expel people. 


Moreover, the simple rules of transparency and reciprocity that gov- 
ern reputation systems generate surprisingly complex results and 
aren't universally welcomed. Moreover, reputation systems will 
become targets as they proliferate and gain power: While power 
may not necessarily corrupt everyone, it does lure the corrupt and 
seduce the powerless who do their bidding. What better means to 
gain advantage than to subvert a reputation system? Even though 
power generally shuns transparency, it can use that transparency to 
make sure people stay in line and say the right things. 


Thus, the transparency of a reputation system doesn't guarantee 
truth in a world where some people wield more power than others. 
Online reputation systems get very, very close to the "real world," 
generating both the transparency and the unreliable agenda-driven 


assertions (aka gossip) that we attribute to villages rather than cities. That trans- 


parency is mostly a guarantee of good behavior, but it can also drive skepticism and 
dissent underground. The challenge then is to build and proliferate reputation sys- 


tems where reputation approximates reality. IIR 1-0 
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Resources & Contact Information 


Dennis Buonafede, buonafede@hotmail.com 


Cory Doctorow, doctorow@craphound.com 


Brian Dear, Birdrock Ventures, brian@birdrock.com; blogs, http://www.brianstorms.com; http://www.nettle.com 


Bud Michael, Biz360, 1 (650) 548-0355 x352, bmichael@biz360.com 

Amjad Hanif, eBay, 1 (408) 376-6332; ahanif@ebay.com 

Brian Burke, eBay, 1 (408) 376-6461; bburke@ebay.com 

Steve Abernethy, SquareTrade, 1 (415) 541-1020; steve@squaretrade.com 

Marc Smith, Microsoft Research, 1 (425) 706-6896; masmith@microsoft.com 

Chris Dellarocas, MIT Sloan School of Management, 1 (617) 258-8115; dell@mit.edu 

Stan Smith, Giorgos Zacharia, OpenRatings, 1 (781) 290-0828; ssmith@openratings.com, 
gzacharia@openratings.com 

Michael Hussey, RateMyTeachers.com, 1 (207) 324-9459; michael@ratemyteachers.com 

Mike Sparger, Rockwell Automation, 1 (414) 382-2000; mfsparger@ra.rockwell.com 

Alexis Johnson, Shopping.com, 1 (650) 616-6544; alexis@epinions.com 

Nirav Tolia, Shopping.com, 1 (650) 616-6555; nirav@shopping.com 

Rob Malda, Slashdot, malda@slashdot.org 

Paul Resnick, University of Michigan, presnick@umich.edu; http://www.si.umich.edu/~presnick/ 


For further reading (and reputation!): 


Reputations Research Network: http://databases.si.umich.edu/reputations 


Affero; Encouraging patronage by offering repuation; www.affero.org 

Elance; eBay for independent contactors; www.elance.com 

Experts Exchange; IT workers share knowledge; www.experts- exchange.com 

FC; Disgruntled employees tell all; www.fuckedcompany.com 

Infirmation; The inside dope on law firms (we'll spare you the jokes); www.infirmation.com 

Kuro5hin; Similar to Slashdot, but not so noisy; www.kuroShin.org 

RateMyProfessors.com; Payback time!; www.ratemyprofessors.com 

*RateMyVCs.com; (We hope to see this, but it may have trouble getting funded!); www.ratemyvcs.com 


Vault; Gruntled and disgruntled employees tell all; www.vault.com 
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OCTOBER 29-31 


NOVEMBER 2-5 


NOVEMBER 3-7 


NOVEMBER 11 


NOVEMBER 14-15 


NOVEMBER 15 


NOVEMBER 17-20 


NOVEMBER 26-28 
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Privacy and Data Security Academy - Chicago, IL. A comprehensive review 
of strategies and standards for privacy and data security. Sponsored by the 
International Association of Privacy Professionals. Register online or call 1 
(800) 266-6501; fax, 1 (215) 545-8107. www.privacyassociation.org/html/con- 
ferences.html 


AeA Classic Financial Conference - San Diego, CA. The American 
Electronics Association sponsors an opporuntity for public technology com- 
panies to showcase their companies to the investment community. To register, 
contact Tina Morais, 1 (408) 987-4234; fax, 1 (408) 727-7057; 
tina_morais@aeanet.org. www.aeanet.org/classic/ 


Next Generation Networks - Boston, MA. The 17th annual conference orga- 
nized by Dave Passmore and John McQuillan will conver the alphabet soup of 
networking, including broadband wireless, IP telephony, network conver- 
gence, storage networking, network security and much more. Call 1 (800) 227- 
1234l; outside the US, call 1 (630) 986-1432; fax 1 (630) 323-5324; e-mail 
registration@bcr.com. www.bcr.com/ngn/ 


Social Networking Shootout - Mountain View, CA. Representatives from 
four social networking companies get grilled on stage by a panel of judges, 
including our own Esther Dyson. Organized and moderated by Rafe 
Needleman. Contact him for details: rafe@rafeneedleman.com. 

website: www.ibdnetwork.com [E] 


Online News Association - Evanston, IL. In its fourth year, the ONA confer- 
ence brings together leading journalists to talk about what’s new in digital 
news. Esther Dyson joins a panel on the future of technology in journalism. 
Register online or call 1 (802) 434-6176; fax, 1 (802) 654-2560; 
HelpDesk@journalists.org www.journalists.org B 


RFID Privacy Workshop - Cambridge, MA. Sponsored in part by MIT, this 
workshop will bring together RFID technologists, boosters, critics, and jour- 
nalists to talk through the privacy issues slowing the spread of RFID technolo- 
gy, as covered in June’s issue. For more information contact Simson L. 
Garfinkel, slg@ex.com. www.rfidprivacy.org 


Comdex - Las Vegas, NV. Still the biggest technology trade show around. This 
year's speakers include Scott McNealy, Bill Gates, Thomas Siebel and John 
Thompson. For more information call Carrie Koeturius, 1 (415) 905-2352; 
fax, 1 (415) 905-2FAX; email, carrie.koeturius@ medialiveinternational.com 
www.comdex.com 


Digital Reykjavik - Reykjavik, Iceland. Explore the potential of a society 
where every home is connected to a fiber network. Speakers include Esther 
Dyson. Register online or call +354 (595) 1500; fax, +354 (595) 1501; registra- 
tion@digitalreykjavik.net. www.digitalreykjavik.com [B 


WWW.EDVENTURE.COM 


Calendar of High-Tech Events 


JANUARY 8-11 


JANUARY 15 


JANUARY 20-23 


FEBRUARY 9-12 


FEBRUARY 15-17 


FEBRUARY 23-26 


FEBRUARY 25-28 


MARCH 21-23 


O Events Esther plans to attend. 


2004 


International CES - Las Vegas, NV. The world's largest consumer technology 
tradeshow. If you're a gadget-head, this is paradise. For more information visit 
the website or call 1 (866) 233-7968; 1 (301) 631-3983 (outside US); fax, 1 
(301) 694-5124; CESinfo@CE.org. www.cesweb.org 


The Churhill Club - Santa Clara, CA. Visionaries and such make theirannual 
top ten technology predictions. Laugh at year’s; believe next year’s? For more 
information, visit the website. www.churchillclub.org B 


LinuxWorld Conference & Expo - New York, NY. The place to be for anyone 
interested in Linux and other open-source technologies. Visit the website or 
call Debbie Diodati 1 (508) 424-48471 or 1 (800) 657-1474 (general 
inquiries); debbie_diodati@idg.com. www.linuxworldexpo.com 


Emerging Technology Conference - San Diego, CA. Organized by O'Reilly & 
Associates, this conference frames the ideas, projects and technologies the 
"alpha geeks" are playing with. Register online or e-mail Linda Holder, lhold- 
er@oreilly.com. conferences.oreillynet.com/etcon E 


Demo 2004 - Scottsdale, AZ. An IDG Executive Forum. Come play with the 
latest gadgets from the newest startups. Register online or contact 1 (800) 643- 
4668 or 1 (508) 460-1385, or via email at registrar@idgexecforums.com. 
www.demo.com 


3GSM World Congress - Cannes, France. The world’s biggest mobile com- 
munications show. Keynotes include Arun Sarin of Vodafone, Rene Obermann 
or T-Mobile and PC Forum 2003 speaker Paul Otellini.Register online or con- 
tact Tamara James, 44 (1932) 893-855; fax, 44 (1932) 893-894; 
cust.serv@informa.com. www.3gsmworldcongress.com 


TED2004 - Monterey, CA. Having presumably covered life and liberty, this 
year’s them is ”The Pursuit of Happiness." An always-eclectic group discusses 
technology, culture, design and other other catalysts for happiness. More 
information online, or contact TED curator Chris Anderson, 1 (650) 292- 
3300; chris@ted.com. www.ted.com 


PC Forum - Scottsdale, AZ. The theme for EDventure’s 27th annual PC 
Forum is “The Big Picture.” The vision is clear; let’s talk about the implemen- 
tation. Stay tuned for details and your invitation. And save the dates! Contact 
Daphne Kis, 1 (212) 924-8800; fax, 1 (212) 924-0240; e-mail, daphne@edven- 
ture.com. www.edventure.com/pcforum/ E 


Lack of a symbol is no indication of lack of merit. The full, current calendar is available on our website, www.edventure.com. 
Please contact Christina Koukkos (christina@edventure.com) to let us know about other events we should include. 
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The conversation never stops! Subscribe to our free e-mail newsletter, The conversation 


continues, for thought-provoking analysis from our editors, along with commentary from our highly intel- 
ligent readers. Sign up at http://www.edventure.com/conversation/join.cfm. 


Release 1.0 Subscription Form 


Complete this form and join the other industry executives who regularly rely on Release 1.0 to stay ahead of the headlines. Or if 


you wish, you can also subscribe online at www.release1-O.com. 


Your annual Release 1.0 subscription costs $795 per year ($850 outside the US, Canada and Mexico), and includes both the print 


and electronic versions of 11 monthly issues; 25% off the cover price when you order from our online archives; a Release 1.0 


binder; the bound transcript of this year’s PC Forum (a $300 value) and an invitation to next year’s PC Forum. 
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